Rasch in R and Quarto, part 1 (updated)

Intro and WAAQ-7

Author
Affiliation
Magnus Johansson
Published

August 16, 2022

1 Why Rasch analysis in R?

  • Traceability
    • Transparency in decisionmaking
  • Reproducibility
  • Freely available
  • Documentation in the same file as analysis
    • (with Quarto/Rmarkdown)

1.1 Agenda overview

  • Sample analysis with provided data and code
    • WAAQ-7
    • PSS-10
  • Suggested structure for analysis
    • Confirmatory and/or exploratory

1.2 Quarto formatting basics

https://quarto.org/docs/authoring/markdown-basics.html

  • Keep the page above open in a web browser
  • We will use library(kableExtra) for tables
    • since it works with HTML/revealjs and PDF output
    • even though library(formattable) looks nicer…

1.3 Exercise 1

  • I’ll walk you through the Rstudio interface
  • You’ll run the setup code chunk, and practice navigation a bit
    • Enable/disable Outline
    • Navigate with Outline
    • Navigate with chunks
    • Switch between source and visual editor
    • Changing size of the quadrant windows
    • Look at the objects df and itemlabels

1.4 Exercise 2

  • We will walk through the code chunks together
  • Learn what they do, but not how (in detail)
  • I’ll explain some basic R concepts as we move ahead
  • Focus on how you can structure the analysis
  • Discuss the output briefly
    • feedback on how to improve is welcome :)
  • Small exercises are described at the end of the file

1.5 Exercise 3 (later on)

  • In the folder “Exercise2” you will find a RaschR2 qmd file
  • It contains a setup block that will import a dataset (PSS-10)
  • You can practice using this file (RaschR1) as a template
    • copy code chunks for your analysis from this file to the other file
    • copy code, run the chunks, check the output, write commentary. Repeat
    • there is a folder called “Exercise2solution” that shows how I did the analysis

2 Background information

The Work-related Acceptance and Action Questionnaire (Bond, Lloyd, and Guenole 2013) is a measure of psychological flexibility contextualized to a workplace setting.

The WAAQ uses 7 questions, all using the same response scale with 7 steps (ranging from “Stämmer aldrig” to “Stämmer alltid”, with labels on all categories).

itemnr item
WAAQ_1 Jag kan arbeta effektiv trots personliga problem som oroar mig
WAAQ_2 Jag kan erkänna mina misstag i arbetet och ändå vara framgångsrik
WAAQ_3 Jag kan arbeta mycket effektivt även när jag är orolig för någonting.
WAAQ_4 Oro hindrar mig inte från att nå framgångar.
WAAQ_5 Jag kan göra vad som krävs, oavsett hur jag mår.
WAAQ_6 Jag kan arbeta effektivt även när jag tvivlar på mig själv.
WAAQ_7 Mina tankar och känslor hindrar mig inte i mitt arbete.

Items in the Swedish WAAQ

2.1 Participants

We have 230 participants and their gender distribution is shown in the table below. All participants are Swedish adults and the dataset was collected to validate a measure of social interactions at workplaces (Johansson 2021; Johansson and Biglan 2021).

Gender distribution in sample
Gender n Percent
Female 76 33.2
Male 153 66.8

2.2 Descriptives of raw data

Responses to all items are summarized below.

Response category Number of responses Percent
0 17 1.1
1 87 5.4
2 150 9.3
3 354 22.0
4 527 32.7
5 387 24.0
6 88 5.5

Total number of responses for all items

2.3 Descriptives - item level

2.3.1 Recoding responses

There are fewer than 10 responses in category 0 for all items, which is problematic for the analysis. We will merge the two lowest categories. For item 2 the four lowest categories will be merged.

Some cells still have fewer than 10 responses. Let’s look at the response categories using Rasch modeling to produce ICC plots.

3 Analysis of response categories

Items 1 and 5 have issues with disordered response categories, which need to be addressed. For both items the two lowest categories will be merged.

3.1 ICC plots for recoded items

Response categories are working as expected, after the adjustments made.

3.2 Floor/ceiling effects - raw data

4 Rasch analysis 1

The eRm package, which uses Conditional Maximum Likelihood (CML) estimation, will be used primarily. For this analysis, the Partial Credit Model will be used.

OutfitMSQ InfitMSQ OutfitZSTD InfitZSTD
WAAQ_1 0.77 0.793 -2.614 -2.379
WAAQ_2 1.371 1.3 3.717 3.158
WAAQ_3 0.667 0.653 -3.881 -4.077
WAAQ_4 0.963 0.912 -0.366 -0.937
WAAQ_5 0.866 0.829 -1.383 -1.995
WAAQ_6 0.849 0.774 -1.66 -2.637
WAAQ_7 0.817 0.803 -2.003 -2.217
PCA of Rasch model residuals
Eigenvalues
1.51
1.38
1.18
1.10
1.01

5 1st contrast loadings

WAAQ_1 WAAQ_2 WAAQ_3 WAAQ_4 WAAQ_5 WAAQ_6 WAAQ_7
WAAQ_1
WAAQ_2 -0.06
WAAQ_3 0.09 -0.24
WAAQ_4 -0.15 -0.06 -0.16
WAAQ_5 -0.13 -0.25 -0.16 -0.16
WAAQ_6 -0.21 -0.18 -0.13 -0.25 -0.12
WAAQ_7 -0.27 -0.28 -0.09 -0.24 -0.06 -0.07
Note:
Relative cut-off value (highlighted in red) is 0.049, which is 0.2 above the average correlation.

itemnr item
WAAQ_1 Jag kan arbeta effektiv trots personliga problem som oroar mig
WAAQ_2 Jag kan erkänna mina misstag i arbetet och ändå vara framgångsrik
WAAQ_3 Jag kan arbeta mycket effektivt även när jag är orolig för någonting.
WAAQ_4 Oro hindrar mig inte från att nå framgångar.
WAAQ_5 Jag kan göra vad som krävs, oavsett hur jag mår.
WAAQ_6 Jag kan arbeta effektivt även när jag tvivlar på mig själv.
WAAQ_7 Mina tankar och känslor hindrar mig inte i mitt arbete.

Values in red are below/above the pre-set cutoff values.

A correlation matrix is created based on the Rasch model residuals for each item. The average correlation is calculated, and item pairs that correlate more than the pre-set cutoff value (see footnote) above the average correlation are indicated in red in the table below.

Items 1 and 3 are correlated above the relative cutoff value. Looking at the item descriptions, both are related to working efficiently when being worried in general (item 3) or being worried about personal problems. As we saw in the section on item fit, item 3 had quite low item fit values. This is a common issue with broad questions.

It would be reasonable to eliminate either question 1 or 3, since they clearly overlap in content. We will leave this for an exercise.

6 DIF analysis

Our sample is too small to conduct a properly powered DIF-analysis of any of the demographic variables available (gender/age). N > 250 per subgroup would be ideal, and N > 150 minimal.

For the sake of practice, we will go through the steps in setting up and conducting a DIF analysis anyway.

We will use the Rasch Tree tool (Strobl, Kopf, and Zeileis 2013), which makes it possible to evaluate interaction DIF, if sample size allows. For this analysis, we will only investigate gender effects on DIF.

[1] "No significant DIF found."

No DIF was found in our small sample.

7 Targeting & item/person locations

7.1 Reliability

“Test information” shows the collective information based on the combination of all items into a unidimensional scale.

The horizontal lines at TIF 3.33 and 5.0 indicate the conventional cutoff values for acceptable reliability of 0.7 and 0.8, respectively.

The table footnote includes information on floor/ceiling effects.

7.1.1 Item information

“Item information” shows individual curves indicating the amount of information contributed by each item.

7.2 Person location and infit ZSTD

8 Item parameters

Threshold 1 Threshold 2 Threshold 3 Threshold 4 Threshold 5 Item location
WAAQ_1 -1.60 -0.58 1.40 4.87 NA 1.02
WAAQ_2 -1.22 0.69 3.86 NA NA 1.11
WAAQ_3 -2.04 -1.67 0.19 2.20 5.81 0.9
WAAQ_4 -1.87 -0.96 -0.10 2.25 4.32 0.73
WAAQ_5 -0.44 0.49 2.33 4.40 NA 1.69
WAAQ_6 -1.43 -0.52 0.33 2.60 5.04 1.2
WAAQ_7 -1.26 -0.75 0.08 1.74 4.44 0.85

9 Ordinal to interval transformation table

Ordinal-interval transformation table
Ordinal sum score Logits SE (Logits) Interval score (0-100) SE (0-100) MU (0-100)
1 -3.45 1.01 0.0 9.7 19.4
2 -2.72 0.73 7.0 7.0 14.0
3 -2.28 0.62 11.2 6.0 12.0
4 -1.94 0.55 14.4 5.3 10.6
5 -1.66 0.51 17.1 4.9 9.8
6 -1.41 0.49 19.5 4.7 9.4
7 -1.18 0.47 21.7 4.5 9.0
8 -0.97 0.46 23.7 4.4 8.8
9 -0.76 0.45 25.7 4.3 8.6
10 -0.56 0.45 27.7 4.3 8.6
11 -0.36 0.45 29.6 4.3 8.6
12 -0.15 0.45 31.6 4.3 8.6
13 0.06 0.46 33.6 4.4 8.8
14 0.28 0.47 35.7 4.5 9.0
15 0.50 0.49 37.8 4.7 9.4
16 0.75 0.50 40.2 4.8 9.6
17 1.01 0.52 42.7 5.0 10.0
18 1.28 0.53 45.3 5.1 10.2
19 1.58 0.55 48.1 5.3 10.6
20 1.89 0.57 51.1 5.5 11.0
21 2.23 0.58 54.4 5.6 11.2
22 2.58 0.60 57.7 5.8 11.6
23 2.95 0.61 61.2 5.9 11.8
24 3.33 0.63 64.9 6.1 12.2
25 3.73 0.64 68.7 6.2 12.4
26 4.16 0.66 72.8 6.4 12.8
27 4.61 0.69 77.1 6.7 13.4
28 5.11 0.74 81.9 7.1 14.2
29 5.72 0.84 87.8 8.1 16.2
31 7.00 NA 100.0 NA NA
Note:
SE = Standard Error. Multiply by 2 for MU = Measurement Uncertainty

10 Item location & infit ZSTD

11 Item location & outfit ZSTD

12 First contrast loadings & item locations

The figure above shows the item loadings on the first contrast factor of the PCA of Rasch model residuals, and the item locations.

13 Software used

Package Version Citation
base 4.2.1 R Core Team (2022)
car 3.1.0 Fox and Weisberg (2019)
cowplot 1.1.1 Wilke (2020)
eRm 1.0.2 Mair and Hatzinger (2007b); Mair and Hatzinger (2007a); Hatzinger and Rusch (2009); Rusch, Maier, and Hatzinger (2013); Koller, Maier, and Hatzinger (2015); Debelak and Koller (2019); Mair, Hatzinger, and Maier (2021)
formattable 0.2.1 Ren and Russell (2021)
grateful 0.1.11 Rodríguez-Sánchez, Jackson, and Hutchins (2022)
HH 3.1.49 Heiberger and Holland (2004); Heiberger and Robbins (2014); Heiberger and Holland (2015); Heiberger (2022)
kableExtra 1.3.4 Zhu (2021)
knitr 1.40 Xie (2014); Xie (2015); Xie (2022)
matrixStats 0.62.0 Bengtsson (2022)
mirt 1.37.1 Chalmers (2012)
mokken 3.0.6 Van der Ark (2007); Van der Ark (2012)
psych 2.2.5 Revelle (2022)
psychotree 0.16.0 Trepte and Verbeet (2010); Strobl, Wickelmaier, and Zeileis (2011); Strobl, Kopf, and Zeileis (2015); Komboz, Zeileis, and Strobl (2018); Wickelmaier and Zeileis (2018)
reshape 0.8.9 Wickham (2007)
RISEkbmRasch 0.1.4 Johansson (2022)
rmarkdown 2.16 Xie, Allaire, and Grolemund (2018); Xie, Dervieux, and Riederer (2020); Allaire et al. (2022)
scales 1.2.1 Wickham and Seidel (2022)
tidyverse 1.3.2 Wickham et al. (2019)

14 Exercises

  1. Make a table of partipant age distribution
  2. Eliminate either item 1 or 3 and check the effects on reliability
    • hint: df.omit.na$item <- NULL
  3. Bonus if you are quick:
    • Run a DIF analysis using age as DIF variable
    • first add the dif.age variable in the setup chunk
    • then duplicate the DIF chunk and modify code

15 References

Allaire, JJ, Yihui Xie, Jonathan McPherson, Javier Luraschi, Kevin Ushey, Aron Atkins, Hadley Wickham, Joe Cheng, Winston Chang, and Richard Iannone. 2022. Rmarkdown: Dynamic Documents for r. https://github.com/rstudio/rmarkdown.
Bengtsson, Henrik. 2022. matrixStats: Functions That Apply to Rows and Columns of Matrices (and to Vectors). https://CRAN.R-project.org/package=matrixStats.
Bond, Frank W., Joda Lloyd, and Nigel Guenole. 2013. “Work-Related Acceptance and Action Questionnaire.” American Psychological Association (APA). https://doi.org/10.1037/t72150-000.
Chalmers, R. Philip. 2012. mirt: A Multidimensional Item Response Theory Package for the R Environment.” Journal of Statistical Software 48 (6): 1–29. https://doi.org/10.18637/jss.v048.i06.
Debelak, Rudolf, and Ingrid Koller. 2019. Testing the Local Independence Assumption of the Rasch Model With Q3-Based Nonparametric Model Tests.” Applied Psychological Measurement. https://doi.org/10.1177/0146621619835501.
Fox, John, and Sanford Weisberg. 2019. An R Companion to Applied Regression. Third. Thousand Oaks CA: Sage. https://socialsciences.mcmaster.ca/jfox/Books/Companion/.
Hatzinger, Reinhold, and Thomas Rusch. 2009. IRT models with relaxed assumptions in eRm: A manual-like instruction.” Psychology Science Quarterly 51.
Heiberger, Richard M. 2022. HH: Statistical Analysis and Data Display: Heiberger and Holland. https://CRAN.R-project.org/package=HH.
Heiberger, Richard M., and Burt Holland. 2004. Statistical Analysis and Data Display: An Intermediate Course with Examples in S-Plus, R, and SAS. First. Springer-Verlag, New York. https://link.springer.com/book/10.1007/978-1-4757-4284-8.
———. 2015. Statistical Analysis and Data Display: An Intermediate Course with Examples in R. Second. Springer-Verlag, New York. https://link.springer.com/book/10.1007/978-1-4939-2122-5.
Heiberger, Richard M., and Naomi B. Robbins. 2014. “Design of Diverging Stacked Bar Charts for Likert Scales and Other Applications.” Journal of Statistical Software 57 (5): 1–32. https://doi.org/10.18637/jss.v057.i05.
Johansson, Magnus. 2021. “Dataset for the Initial Development of the Group Nurturance Inventory.” figshare. https://doi.org/10.6084/M9.FIGSHARE.13042010.V3.
———. 2022. RISEkbmRasch: Psychometric Analysis with Rasch Measurement Theory. https://github.com/pgmj/RISEkbmRasch.
Johansson, Magnus, and Anthony Biglan. 2021. “The Group Nurturance Inventory Initial Psychometric Evaluation Using Rasch and Factor Analysis.” BMC Public Health 21 (1). https://doi.org/10.1186/s12889-021-11474-5.
Koller, Ingrid, Marco Johannes Maier, and Reinhold Hatzinger. 2015. An Empirical Power Analysis of Quasi-Exact Tests for the Rasch Model: Measurement Invariance in Small Samples.” Methodology 11. https://doi.org/10.1027/1614-2241/a000090.
Komboz, Basil, Achim Zeileis, and Carolin Strobl. 2018. “Tree-Based Global Model Tests for Polytomous Rasch Models.” Educational and Psychological Measurement 78 (1): 128–66. https://doi.org/10.1177/0013164416664394.
Mair, Patrick, and Reinhold Hatzinger. 2007a. CML based estimation of extended Rasch models with the eRm package in R.” Psychology Science 49.
———. 2007b. Extended Rasch modeling: The eRm package for the application of IRT models in R.” Journal of Statistical Software 20. https://www.jstatsoft.org/v20/i09.
Mair, Patrick, Reinhold Hatzinger, and Marco Johannes Maier. 2021. eRm: Extended Rasch Modeling. https://cran.r-project.org/package=eRm.
R Core Team. 2022. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Ren, Kun, and Kenton Russell. 2021. Formattable: Create ’Formattable’ Data Structures. https://CRAN.R-project.org/package=formattable.
Revelle, William. 2022. Psych: Procedures for Psychological, Psychometric, and Personality Research. Evanston, Illinois: Northwestern University. https://CRAN.R-project.org/package=psych.
Rodríguez-Sánchez, Francisco, Connor P. Jackson, and Shaurita D. Hutchins. 2022. Grateful: Facilitate Citation of r Packages. https://github.com/Pakillo/grateful.
Rusch, Thomas, Marco Johannes Maier, and Reinhold Hatzinger. 2013. Linear logistic models with relaxed assumptions in R.” In Algorithms from and for Nature and Life, edited by Berthold Lausen, Dirk van den Poel, and Alfred Ultsch. Studies in Classification, Data Analysis, and Knowledge Organization. New York: Springer. https://doi.org/10.1007/978-3-319-00035-0_34.
Strobl, Carolin, Julia Kopf, and Achim Zeileis. 2013. “Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model.” Psychometrika 80 (2): 289–316. https://doi.org/10.1007/s11336-013-9388-3.
———. 2015. “Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model.” Psychometrika 80 (2): 289–316. https://doi.org/10.1007/s11336-013-9388-3.
Strobl, Carolin, Florian Wickelmaier, and Achim Zeileis. 2011. “Accounting for Individual Differences in Bradley-Terry Models by Means of Recursive Partitioning.” Journal of Educational and Behavioral Statistics 36 (2): 135–53. https://doi.org/10.3102/1076998609359791.
Trepte, Sabine, and Markus Verbeet, eds. 2010. Allgemeinbildung in Deutschland – Erkenntnisse Aus Dem SPIEGEL Studentenpisa-Test. Wiesbaden: VS Verlag.
Van der Ark, L. Andries. 2007. “Mokken Scale Analysis in R.” Journal of Statistical Software 20 (11): 1–19. https://www.jstatsoft.org/article/view/v020i11.
———. 2012. “New Developments in Mokken Scale Analysis in R.” Journal of Statistical Software 48 (5): 1–27. https://www.jstatsoft.org/article/view/v048i05.
Wickelmaier, Florian, and Achim Zeileis. 2018. “Using Recursive Partitioning to Account for Parameter Heterogeneity in Multinomial Processing Tree Models.” Behavior Research Methods 50 (3): 1217–33. https://doi.org/10.3758/s13428-017-0937-z.
Wickham, Hadley. 2007. “Reshaping Data with the Reshape Package.” Journal of Statistical Software 21 (12). https://www.jstatsoft.org/v21/i12/.
Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the tidyverse.” Journal of Open Source Software 4 (43): 1686. https://doi.org/10.21105/joss.01686.
Wickham, Hadley, and Dana Seidel. 2022. Scales: Scale Functions for Visualization. https://CRAN.R-project.org/package=scales.
Wilke, Claus O. 2020. Cowplot: Streamlined Plot Theme and Plot Annotations for ’Ggplot2’. https://CRAN.R-project.org/package=cowplot.
Xie, Yihui. 2014. “Knitr: A Comprehensive Tool for Reproducible Research in R.” In Implementing Reproducible Computational Research, edited by Victoria Stodden, Friedrich Leisch, and Roger D. Peng. Chapman; Hall/CRC. http://www.crcpress.com/product/isbn/9781466561595.
———. 2015. Dynamic Documents with R and Knitr. 2nd ed. Boca Raton, Florida: Chapman; Hall/CRC. https://yihui.org/knitr/.
———. 2022. Knitr: A General-Purpose Package for Dynamic Report Generation in r. https://yihui.org/knitr/.
Xie, Yihui, J. J. Allaire, and Garrett Grolemund. 2018. R Markdown: The Definitive Guide. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/rmarkdown.
Xie, Yihui, Christophe Dervieux, and Emily Riederer. 2020. R Markdown Cookbook. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/rmarkdown-cookbook.
Zhu, Hao. 2021. kableExtra: Construct Complex Table with ’Kable’ and Pipe Syntax. https://CRAN.R-project.org/package=kableExtra.